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SUBJECT:  Audit  Report  on  Use  of  Foreign  Materiel  Exploitation  Results 
(Report  No.  98-005) 


We  are  providing  this  audit  report  for  information  and  use.  We  considered 
management  comments  on  a  draft  of  this  report  in  preparing  the  final  report. 

Conunents  on  the  draft  of  this  report  conformed  to  the  requirements  of  DoD 
Directive  7650.3  and  left  no  unresolved  issues.  Therefore,  no  additional  comments  are 
required. 


We  appreciate  the  courtesies  extended  to  the  audit  staff.  Questions  on  the 
audit  should  be  directed  to  Mr.  Robert  K.  West,  Audit  Program  Director,  at  (703)  604- 
9804  (DSN  664-9804)  or  Mr.  Robert  W.  Often,  Audit  Project  Manager,  at  (703)  604- 
9815  (DSN  664-9815).  See  Appendix  B  for  the  report  distribution.  The  audit  team 
members  are  listed  inside  the  back  cover. 


Robert  Lieberman 
Assistant  Inspector  General 
for  Auditing 
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Report  No.  98-005  October  8,  1997 

(Project  No.  7 AD-0006) 

Use  of  Foreign  Materiel  Exploitation  Results 


Executive  Summary 


Introduction.  Foreign  materiel  exploitation  involves  analysis,  testing,  and  evaluation 
of  foreign  materiel,  to  include  testing  against  U.S.  equipment.  Foreign  materiel 
exploitation  supports  DoD  acquisition  programs,  testing,  toeat  simulator  and  target 
development,  modeling  and  simulation,  and  training  and  tactics.  For  FY  1996,  Ae 
Military  Departments  estimated  that  they  spent  about  $117  million  on  foreign  materiel 
exploitation  projects. 

Audit  Objective .  The  audit  objective  was  to  evaluate  the  timeliness  and  effectiveness 
of  the  coordination,  dissemination,  and  use  of  foreign  materiel  exploitation  results. 
Specifically,  we  reviewed  how  the  Military  Departments  used  foreign  materiel 
exploitation  results  for  modifying  simulators,  for  assisting  in  target  development,  and 
for  training  and  tactics  purposes.  We  also  reviewed  the  management  control  program 
related  to  ttiat  objective. 

Audit  Results.  The  DoD  Components'  scientific  and  technical  intelligence  centers 
coordinated  and  disseminated  foreign  materiel  exploitation  results  to  the  test  and 
evaluation,  simulator  and  target  development,  and  training  and  tactics  communities 
effectively  and  in  a  timely  manner.  However,  the  Military  Departments  did  not  always 
validate  simulated  threat  systems  used  to  support  developmental  and  operational  test 
and  evaluation  based  on  the  latest  foreign  materiel  exploitation  data.  As  a  result,  threat 
models  and  simulators  used  to  support  developmental  and  operational  test  and 
evaluation  may  not  accurately  represent  the  real  world  threat.  For  details  on  the  audit 
results,  see  Part  I. 

Summary  of  Recommendations.  We  reconunend  that  the  Navy  develop  a  Navy-wide 
validation  process  and  that  the  Air  Force  revise  its  validation  guidance  to  include 
criteria  for  determining  when  to  revalidate  models  and  threat  simulators. 

Management  Comments.  The  Navy  concurred  with  the  finding  and  recommendation 
and  stated  that  it  would  develop  a  Navy-wide  validation  process.  The  Air  Force 
nonconcurred  on  the  basis  that  we  did  not  recognize  its  ongoing  effort  to  revise  Air 
Force  Instruction  16-1001,  "Verification,  Validation,  and  Accreditation,"  June  1,  1996, 
which  Air  Force  officials  believe  meets  the  intent  of  our  audit  recommendation.  See 
Part  I  for  a  discussion  of  the  Navy  and  Air  Force  conunents,  and  see  Part  III  for  the 
complete  text  of  the  comments. 

Audit  Response.  The  Navy  and  Air  Force  comments  are  fully  responsive;  therefore, 
additional  comments  are  not  required.  We  revised  the  final  report  to  recognize  the 
Air  Force  effort  to  revise  Air  Force  Instruction  16-1001.  The  criteria  for  when  to 
revalidate  models  and  threat  simulators  were  not  in  the  Air  Force  draft  instruction  that 
we  reviewed  during  the  audit.  However,  the  criteria  were  in  an  improved  draft  version 
of  Air  Force  Instruction  16-1001  that  the  Air  Force  provided  with  its  response  to  our 
draft  audit  report.  The  Air  Force  plans  to  publish  the  revised  instruction  by 
December  31,  1997. 
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Part  I  -  Audit  Results 


Audit  Background 


Foreign  materiel  exploitation  involves  analysis,  testing,  and  evaluation  of 
foreign  materiel,  to  include  testing  against  U.S.  equipment.  Foreign  materiel 
exploitation  supports  DoD  acquisition  programs,  testing,  threat  simulator  and 
target  development,  modeling  and  simulation,  and  training  and  tactics.  For 
FY  1996,  the  h^t^  Departments  estimated  that  they  spent  about  $117  million 
on  foreign  materiel  exploitation  projects. 


Audit  Objective 


The  audit  objective  was  to  evaluate  the  timeliness  and  effectiveness  of  the 
coordination,  dissemination,  and  use  of  foreign  materiel  exploitation  results. 
Specifically,  we  reviewed  how  tiie  Milit^  Departments  used  foreign  materiel 
exploitation  results  for  modifying  simulators,  for  assisting  in  target 
development,  and  for  training  and  tactics  purposes.  We  also  reviewed  tfie 
management  control  program  related  to  that  objective.  Appendix  A  discusses 
our  scope,  methodology,  and  management  control  program  review  results. 
Appendix  A  also  discusses  a  summary  of  prior  coverage  related  to  our  audit 
objective. 


Coordination  and  Dissemination  of  Exploitation  Results 


The  DoD  Components'  scientific  and  technical  intelligence  centers  coordinated 
and  disseminate  foreign  materiel  exploitation  results  to  the  test  and  evaluation, 
simulator  and  target  development,  and  training  and  tactics  communities 
effectively  and  in  a  timely  manner. 

Missile  and  Space  Uitelligence  Center.  The  Missile  and  Space  Intelligence 
Center  coordinated  and  disseminated  exploitation  results  efficiently  md 
effectively  by  conducting  symposiums,  issuing  electronic  messages,  issuing 
interim  rqxjrts  after  each  exploitation  phase,  and  issuing  rqports  whai 
foreign  materiel  exploitation  was  completed.  The  Missile  and  Space 
Inteltigence  Center  disseminated  those  messages  and  rqiorts  throughout  the 
exploitation  process  to  the  DoD  Components. 

Army.  The  National  Ground  Intelligence  Center  disseminated  foreign  materiel 
exploitation  results  on  foreign  ground  systems  and  helicopters  through  written 
test  and  foreign  materiel  exploitation  reports,  electronic  and  magnetic  me^, 
data  bases,  photos,  and  videos.  The  Center  iso  published  a  list  of  all  foreign 
materiel  exploitation  reports  issued  by  the  Center  and  other  organizations  each 
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quarter.  Furthermore,  the  Center  incorporated  foreign  materiel  oploitalion 
results  into  the  Defense  Intelligence  Agency  Intelligence  Reduction  Program. 
Specifically,  the  National  Ground  Intelligence  Center  assign^  an  intelligence 
analyst  to  eve^  foreign  sjrstem  under  tiie  Center's  executive  agency.  The 
analysts  compifed  information  on  foreign  systems  that  the  Carter  obtained 
through  foreign  materiel  exploitation  and  other  intelligence  sources.  The 
Center's  analysts  maintained  and  updated  the  data  on  foreign  threat  systems  for 
the  Defense  Intelligence  Agency  and  made  that  information  avaiMle  to  all 
interested  users  without  attributing  the  data  to  any  specific  source. 

Navy.  The  Office  of  Naval  Intelligence  disseminated  foreign  materiel 
exploitation  results  tiurough  foreign  materiel  exploitation  rqrorts,  though 
message  tiaffic,  and  through  the  Defense  Intelligence  Agency  Intelligence 
Production  Program.  The  Office  of  Naval  Intelligence  also  publiimed  an  annual 
foreign  materiel  exploitation  rg)ort.  That  report  provides  a  description  and 
status  on  all  Navy  foreign  materiel  exploitation  projects. 

Air  Force.  The  National  Air  fiitelligence  Center  coordinated  and  disseminated 
exploitation  results  effectively  and  efficiently  through  electronic  messages, 
interim  and  final  rqwrts,  and  the  Defense  Intelligence  Agency  Intelligent 
Production  Program.  The  Center  also  published  a  foreign  matoiel  triennial 
status  report  and  a  weekly  r^rt  on  aerospace  systems.  Both  rqwrts  include 
foreign  materiel  exploitation  results.  In  addition,  the  Air  Force  Test  and 
Evaluation  Directorate,  Foreign  Materiel  Office,  had  an  ongoing  ini^tiye  to 
involve  contractors  early  in  &e  exploitation  process.  Although  the  initiative 
pertained  only  to  electronic  warfare  systems,  the  Air  Force  had  plans  to  expand 
the  program  to  otiier  systems  dq>ending  on  the  successes  and  lessons  learned 
from  that  program.  The  >rir  Force  shared  the  initiative  with  the  Missile  and 
Space  Intelligence  Center  and  tee  Military  D^artments.  If  successful,  tee 
Missile  and  Space  Intelligence  Center  and  tee  Military  Dq)artments  could  also 
use  tee  concept  in  their  exploitation  process. 


Validation  of  Simulated  Threat  Systems 

The  Military  Departments  did  not  always  validate  simulated  threat 
systems  used  to  support  developmental  and  operational  test  and 
evaluation  based  on  the  latest  foreign  materiel  exploitation  data. 
However,  the  Army  was  taking  action  to  improve  its  threat  validation 
program  by  prioritizing  threat  validation  requirements.  The  Navy  did 
not  develop  a  common  Navy-wide  vali^tion  and  revalidation  process  for 
threat  systems,  and  the  Air  Force  did  not  have  criteria  on  when  to 
revalidate  threat  systems.  As  a  result,  threat  models  and  simulators  used 
to  support  developmental  and  operational  test  and  evaluation  may  not 
accurately  r^resent  the  real  world  threat. 


DoD  Model  and  Simulation  Validation  Policy  and  Procedures 


DoD  Validation  Guidance.  DoD  Regulation  S0(X).2-R,  "Mandatory 
Procedures  for  Major  Defense  Acquisition  Programs  (MDAPs)  and  Major 
Automated  Information  Systems  (N^S)  Acquisition  Programs,"  March  15, 
1996,  requires  that  each  representation  of  a  threat  model  or  simulation  be 
subjected  to  validation  procedures.  DoD  Instruction  5000.61,  "DoD  Modeling 
and  Simulation  (M&S)  Verification,  Validation,  and  Accreditation  (W&A)," 
April  29,  1996,  implements  policy,  assigns  responsibility,  and  prescribes 
procedures  for  validation  of  DoD  models  and  simulations.  That  instruction 
states  that  the  DoD  Components  shall  establish  validation  policies  and 
procedures  for  models  and  simulations.  The  DoD  Executive  Council  for 
Modeling  and  Simulations  recommends  DoD-wide  modeling  and  simulation 
goals  and  objectives  to  tiie  Under  Secretary  of  Defense  for  Acquisition  and 
Technology. 

DoD  Threat  Simulator  Program  Plan.  The  DoD  Threat  Simulator  Program 
Plan,  February  1993,  states  tiiat  threat  simulators  that  DoD  Components  use  as 
test  resources  are  subject  to  DoD  approved  validation  procedures.  The  DoD 
Threat  Systems  Office  is  revising  that  plan  to  u^te  threat  validation 
responsibilities  and  policies  and  to  extend  the  scope  of  the  DoD  threat  validation 
program  to  include  targets,  digital  models,  and  simulations.  The  DoD  Threat 
Systems  Office  reviews  DoD  Components'  validation  rqwrts  for  the  DoD 
Threat  Systems  Validation  Committee  and  provides  indq)endent  ov^sight  of 
and  manages  all  activities  related  to  the  DoD  threat  systems  program  for  the 
Under  Secretary  of  Defense  for  Acquisition  and  Technology.  The  Deputy 
Director,  Defense  Test,  Systems  Engineering,  and  Evaluation/Test  Facility 
Resources,  chairs  the  DoD  Threat  Systems  Validation  Committee.  The  Deputy 
Director,  supported  by  the  Committee,  is  responsible  for  approving  DoD 
Components'  v^dation  reports. 


"^e  DoD  threat  systems  program  includes  actu^  threat  hardware,  threat 
simulators,  threat  targets,  threat  models,  and  threat  simulations. 
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Validation  of  Simulated  Threat  Systems 


Threat  Validation.  Validation  is  the  process  of  determining  the  extent  to 
which  models  and  simulations  accurately  rq)resent  the  real  world  from  the 
perspective  of  the  intended  user.  For  test  and  evaluation  usen,  threat  validation 
is  the  identification,  analysis,  and  documentation  of  the  differences  between 
models,  simulators,  and  targets  (simulated  threat  systems)  and  toe  threats  that 
toey  rq)resent  so  that  differences  can  be  factored  into  test  planning,  execution, 
and  evaluation. 

nireat  Validation  Responsibility.  Each  Service  is  responsible  for  thrpt 
validation  of  its  own  simulated  toeat  systems  and  for  preparing  validation 
rqwrts  that  document  toe  differences  between  toe  simulated  threat  systems  and 
toe  corresponding  actual  threats. 

Test  Accreditation.  Test  accreditation  is  toe  official  certification  that  a 
simulated  threat  system  is  accq)table  for  use  in  a  specific  test.  Thr^t  validation 
results  provide  DoD  developmental  and  operation^  test  and  evaluation  users  toe 
information  that  they  need  to  help  make  that  determination. 


Army  Threat  Validation 


Army  Threat  Validation  Guidance.  Army  Regulation  5-11,  "Army  Model 
and  Simulation  Management  Program,"  June  10,  1992,  establishes  toe  Anny 
Model  and  Simulation  M^agement  Program.  That  regulation  also  assigns 
responsibilities  and  prescribes  policy  and  guidance  for  toe  execution  of  toe 
program.  The  "U.S.  Army  Validation  and  Accreditation  Plan  for  Threat 
Simulators  and  Targets,"  November  1993,  lists  specific  Army  threat  validation 
and  test  accreditation  procedures.  The  Army  is  revising  that  plan  to  reflect  toe 
experience  gained  tlnough  implementation  of  toe  threat  validation  process.  The 
Army  plans  to  publish  toe  revised  plan  in  July  1997  as  part  of  Dq)artment  of 
toe  Army  Pamphlet  73-1,  "Test  and  Evaluation  in  Support  of  System 
Acquisition." 

Army  Threat  Validation  Process.  Validation  working  groups  validate  specific 
simulated  threat  systems.  The  Army  Test  and  Evaluation  Management  Agency 
chairs  Army  validation  working  groups.  Other  members  of  validation 
working  groups  include  toe  Army  Operational  Test  and  Evaluation  Commandj 
toe  appropriate  Scientific  and  Technical  Intelligence  Center;  toe  Project 
Manager-Instrumentation,  Targets,  and  Threat  Simulators  (PM-iTiS);  and  toe 
threat  system  developer.  Other  members  may  include  representatives  from  toe 
acquisition,  research  and  development,  and  trainin|  communities,  if  needed. 
The  vali^tion  working  group  prepares  a  validation  report.  TEMA  toen 
approves  and  submits  toe  rqiort  to  toe  Threat  Systems  Validation  Committee 
through  toe  DoD  Ibreat  Systems  Office. 

Army  Threat  Validation.  The  Army  validated  9  of  13  simulated  tiueat 
systems,  listed  in  TEMA  records,  needed  to  support  FYs  1995  through  1998 
test  and  evaluation  efforts.  Foreign  materiel  exploitation  data  were  available  for 
8  of  toe  13  systems  before  toe  systems  were  to  be  used  to  support  test  and 
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Validation  of  Simulated  Threat  Systems 


evaluation  efforts.  Of  those  eight  systems  for  which  foreign  materiel 
exploitation  results  were  available,  the  Army  validated  six  of  the  systems  and 
scheduled  one  of  the  systems  for  validation  during  FY  1998.  Also,  the  Army 
validated  three  of  the  five  systems  for  which  foreign  materiel  exploitation  data 
were  not  available. 

Threat  Validation  Waivers.  Army  officials  stated  that  TEMA  waived  threat 
validation  requirements  for  some  simulated  threat  systems  because  of  time 
constraints  or  very  limited  requirements  for  use  of  the  systems.  Army  officials 
also  stat^  that  in  those  situations,  test  accreditation  could  substitute  for  threat 
validation.  Specifically,  simulated  threat  systems  caimot  be  used  to  support  any 
developmental  or  operational  test  until  the  Army  accredits  the  system.  Li 
accrediting  a  system  for  use  in  a  test,  a  Threat  Accreditation  Working  Group 
performs  a  limited  validation  of  the  system  by  comparing  the  characteristics  of 
the  simulated  threat  system  to  be  observed  or  measured  against  the  actual  threat. 
Based  on  that  comparison,  the  Threat  Accreditation  Working  Group  decides 
whether  or  not  to  accredit  the  simulated  threat  system  for  use  in  the  test. 
Representatives  from  Ae  acquisition,  test  and  evaluation,  training,  ^d 
intelligence  communities  are  memb^  of  the  Threat  Accreitation  Working 
Group.  Although  test  accreditation  do^  not  compare  all  of  the  simulated 
characteristics  against  the  actual  characteristics  of  the  threat  system,  the  Army 
maintains  that  developmental  and  operational  test  accreditation  rasures  that 
simulated  threat  conditions  are  very  similar  to  real  world  threat  conditions. 

Threat  Validation  Priority.  The  Army  was  taking  steps  to  improve  its  threat 
validation  program  by  prioritizing  its  threat  validation  requirements.  In  a 
March  1997  memorandum  to  Headquarters,  Army  Materiel  Command, 
PM-ITTS  issued  a  prioritized  list  of  the  simulator  validation  needs  of  the  Army. 
The  Army  assigned  the  first  priority  to  threat  simulators  that  the  Army  plannM 
to  use  to  support  upcoming  testing.  The  Army  assigned  the  next  priority  to 
simulators  that  the  Army  may  use  to  support  testing.  If  a  short-term  testing 
requirement  for  a  simulator  occurs,  the  Amy  would  accredit  it  for  that  specific 
test.  Ihe  Army  did  not  plan  to  validate  simulators  that  it  did  not  plan  to  use  to 
support  future  developmental  and  operational  testing. 

Threat  Validation  Planning.  In  connection  with  that  effort,  PM-ITTS  hosted 
a  March  1997  validation  planning  meeting  to  review  the  status  and  priority  of 
all  simulators  pending  threat  validation.  Representatives  from  the  Army 
Operational  Test  and  Evaluation  Command,  PM-ITTS,  and  TE^  discussed 
upcoming  threat  validation  schedules,  validation  reporting  r^uirements,^  and 
foreign  materiel  exploitation  requirements.  The  meeting  participants  established 
a  goal  to  align  planned  validations  with  simulated  threat  systems  developmental 
schedules.  Because  of  the  TEMA  and  PM-ITTS  initiatives  discussed  above,  we 
are  not  making  any  recommendations  to  the  Army. 
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Navy  Threat  Validation 


Navy  Validation  Guidance.  Secretary  of  the  Navy  and  Chief  of  Naval 
(^rations  guidance  prescribes  Navy  v^dation  policies  and  procedures.  In 
addition,  the  Naval  Air  Systems  Command  (NAVAIR)  has  criteria  for  threat 
validation  of  systems  under  its  control. 

Secretary  of  the  Navy  Validation  Guidance.  Office  of  die  Secretary 
of  the  Navy  Instruction  5200.38,  "Department  of  the  Navy  Modeling  and 
Simulation  Program,"  October  18,  1994,  requires  the  Navy  Modeliiig  s^d 
Simulation  Management  Office  to  write  and  coordinate  a  common  validation 
process  for  all  Navy  organizations. 

Office  of  the  Chief  of  Naval  Operations  Guidance.  Office  of  the 
Chief  of  Naval  Operations  Instruction  3960.15,  "Validation  of  Navy  Air 
Defense  Threat  Simulators,"  May  6,  1991,  assigns  NAVAIR  as  the  Chief  of 
Naval  Operations  technical  agent  for  Navy  air  defense  threat  validation  and 
tasks  NAVAIR  to  implement  validation  procedures. 

NAVAIR  Guidance.  Commander,  Naval  Air  Systems  Command, 
Instruction  3960,  "Validation  of  Navy  Air  Defense  Threat  Simulators,"  Mjw  4, 
1992,  implements  policy  and  assigns  responsibilities  for  the  validation  of  air 
defense  simulators  being  develop^  or  used  for  development  and  testing  of 
airborne  weapon  systems  or  training  of  Navy  aircrew  personnel.  Specifically, 
NAVAIR  is  responsible  for  the  funding  for  the  Naval  Air  Warfare  Center 
Weapons  Division,  China  Lake.  The  Naval  Air  Warfare  Center  established  an 
independent  simulator  validation  coordinator  for  NAVAIR  simulator  validation 
at  all  Navy  laboratories  and  ranges.  The  Navy  Air  Defense  Threat  Simulator 
Validation  Procedures  Manual,  March  1995,  established  the  procedures  to 
validate  Navy  air  defense  and  air  defense-related  threat  simulators  used  for  test 
and  evaluation  and  aircrew  training. 

NAVAIR  Threat  Systems  Validation.  NAVAIR  did  not  validate  all  of  its 
simulators.  NAVAIR  used  and  maintained  41  operational  threat  simulators  at 
the  Naval  Air  Warfare  Center  Weapons  Division,  China  Late.  NAVAIR 
validated  11  of  the  41  simulators  and  ^heduled  4  simulators  for  validation  in 
FY  1998.  New  foreign  materiel  exploitation  data  received  after  the  validation 
of  2  of  the  11  systems  showed  that  those  2  systems  no  longer  represent  the 
current  threat.  However,  NAVAIR  started  to  update  one  of  the  two  systems, 
which  will  be  completed  in  FY  1998. 

Navy  Threat  Validation  Process.  The  Navy  did  not  have  a  standard  policy  for 
validation  of  Navy  models  and  simiilators  because  the  Navy  Modeling  and 
Simulation  Management  Office  did  not  develop  a  common  Navy-vwde  validation 
process  as  Office  of  the  Secretary  of  the  Navy  Instruction  5200.38, 
"Department  of  the  Navy  Modeling  and  Simulation  Program,"  October  18, 
1994,  requires.  As  a  result.  Navy  organizations  validate  models  and  simulators 
differently.  That  practice  could  result  in  Navy  threat  simulators  not 
representing  the  most  current  threat  capabilities  and  characteristics  of  foreign 
materiel  acquired  and  exploited. 
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Validation  of  Simulated  Threat  Systems 


Mine  and  Torpedo  Simulator  Validation.  The  Naval  Undersea  Warf^ 
Center  and  the  Naval  Coastal  Systems  Station  did  not  develop  procedur^  for 
foreign  mine  and  torpedo  simulator  validation.  Without  validation  procures, 
the  potential  exists  that  mine  and  torpedo  simulators  may  not  r^resent  the  most 
current  capabilities  and  characteristics  of  the  foreign  materiel  acquired  and 
exploited.  That  situation  could  impact  the  program  manager's  ability  to  make 
sound  milestone  decisions  in  support  of  acquisition  programs  and  m  the 
development  of  tactics  and  countermeasures. 


Air  Force  Threat  Validation 


Air  Force  Validation  Guidance.  Air  Force  Instruction  16-1001,  "Verification, 
Validation,  and  Accreditation,"  June  1,  1996,  establishes  poHcy,  pr^i^, 
and  responsibilities  for  the  validation  of  Air  Force  owned  or  managed  models 
and  simulations.  In  addition,  the  Air  Force  was  drafting  an  attachment  to  that 
instruction  to  specificalW  address  target,  threat  simulator,  and  threat  models  and 
giimiiatinns  and  veimcation,  validation,  and  accreditation  for  test  and 
evaluation.  Although  the  instruction  states  that  validation  will  be  a  continuous 
process  throughout  the  model  and  simulation  life  cycle,  neither  the  instruction 
nor  the  draft  attachment  provided  criteria  for  determining  when  the  Air  Force 
should  revalidate  models  and  threat  simulators.  We  expressed  our  concerns  to 
officials  from  the  Simulation  Validation  Office,  National  Air  Intel^OTce 
Center,  and  the  Chief,  Air  Force  Modeling,  Simulation,  and  Analysis  (Poucy 
and  Standards  Division),  and  those  officials  agreed  mth  our  position.  After 
completion  of  our  au^t  field  work,  the  Air  Force  had  included  the  criteria  m  a 
revised  draft  of  Air  Force  Instruction  16-1001.  The  Air  Force  planned  to 
publish  the  revised  instruction  by  December  31,  1997. 

Air  Force  Threat  Validation  Process.  The  Simulation  Validation  Office, 
National  Air  Intelligence  Center,  performs  threat  simulator  validations  and 
prepares  validation  reports  that  it  submits  to  the  Crossbow  Committee  through 
the  DoD  Threat  Systems  Office. 

Air  Force  Threat  Validation.  Air  Force  test  and  range  personnel  stated  that 
the  Air  Force  generally  revalidated  threat  system  simulators  when  the  simulator 
developer  or  other  users  changed  or  momfied  the  system.  Users  of  threat 
system  simulators  need  to  be  aware  of  the  differences  between  simulator 
performance  and  the  current  threat  system  capability  to  detomine  whether  the 
tiireat  simulator  can  meet  the  test  requirements.  The  Air  Force  should  also 
revalidate  fielded  simulators  used  to  support  test  and  evaluation  when  new  threat 
data  become  available. 
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Validation  of  Simulated  Threat  Systems 


Recommendations,  Management  Comments,  and  Audit 
Response 


1.  We  recommend  that  the  Director,  Navy  Modeling  and  Simulation 
Minagement  OUice,  develop  a  Navy-wide  validation  process  for  all  Navy 
organizations  in  accordance  with  the  Office  of  the  Secretary  of  the  Navy 
lostmction  5200.38,  "Department  of  the  Navy  Modeling  and  Simulation 
Program,”  October  18, 1994. 

Navy  Comments.  The  Navy  concurred  with  tiie  finding  and  recommendation 
and  stated  that  it  would  develop  a  Navy-wide  validation  process  in  accordance 
with  the  Office  of  Ae  Secretary  of  the  Navy  Instruction  5200.38. 

2.  We  recommend  that  the  Chief,  Air  Force  Modeling,  Simulation,  and 
Analysis  (Policy  and  Standards  Division),  Directorate  of  Command  mid 
Control,  revise  Air  Force  Instruction  1^1001,  "Verification,  V^dation, 
and  Accreditation,"  June  1,  1996,  to  include  criteria  for  ^termimng  when 
to  revalidate  models  and  threat  simulators.  The  criteria  should  include 
changes  in  threat  data  and  dianges  or  modifications  to  threat  system 
simu&tors. 

Air  Force  Comments.  The  Air  Force  nonconcurred  with  our  finding  and 
stated  that  we  did  not  recognize  the  ongoing  effort  of  the  Air  Force  to  revise 
Air  Force  Instruction  16-1001.  The  Air  Force  stated  that  it  was  concluding  a 
7-month  effort  to  revise  Ae  Air  Force  instruction,  which  woidd  include  the 
recommended  criteria.  The  Air  Force  planned  to  publish  the  revised  instruction 
by  December  31,  1997. 

Audit  Response.  The  Air  Force  comments  are  fully  responsive;  thaefore,  no 
additional  comments  are  required.  We  revised  the  fmal  report  to  recognize  the 
Air  Force  effort  to  revise  Air  Force  Instruction  16-1001.  The  criteria  for 
determining  when  to  revalidate  models  and  threat  simulators  were  not  in  the 
dr^  instruction  that  we  reviewed  during  the  audit.  However,  the  critoia  were 
included  in  the  draft  version  diat  the  Air  Force  provided  us  wiA  its  comments  to 
tWs  report. 
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Part  n  -  Additional  Information 


Appendix  A.  Audit  Process 


Scope  and  Methodology 


This  program  audit  was  conducted  from  October  1996  through  May  1997  in 
accordance  with  auditing  standards  issued  by  the  Comptroller  Genei^  of  the 
United  States  as  implemented  by  die  Inspector  General,  DoD.  Accordingly,  we 
included  tests  of  management  controls  considered  necessary.  The  scope  of  our 
review  covered  the  Office  of  the  Secret^  of  Defense,  the  Military 
Departments,  the  Defense  Modeling  and  Simulation  Office,  the  Defense 
Intelligence  Agency,  and  DoD  contr^tors.  We  interviewed  cognizant  DoD, 
Military  Dqtartment,  Defense  Intelligence  Agency,  and  contractor  officials. 
We  reviewed  and  ev^uated  documentation  dat^  from  November  1993  through 
May  1997.  We  did  not  use  computer-processed  data  or  statistical  sampling 
procedures  for  this  audit. 

We  visited  or  contacted  individuals  and  organizations  within  the  DoD  and 
Army,  Navy,  and  Air  Force  foreign  materiel  exploitation  contractors.  Further 
det^s  are  available  on  request. 


Management  Control  Program 


DoD  Directive  5010.38,  "Management  Control  (MC)  Program,"  August  26, 
1996,  requires  that  DoD  organizations  implement  a  comprehensive  system  of 
management  controls  that  provides  reasonable  assurance  that  programs  are 
operating  as  intended  and  to  evaluate  the  adequacy  of  the  controls. 

Scope  of  Review  of  Management  Control  Program.  We  reviewed  the 
adequacy  of  DoD  Components'  managemrat  controls  over  the  timeliness  and 
effectiveness  of  the  coordination,  dissemination,  and  use  of  foreign  materiel 
exploitation  results.  Because  we  did  not  identify  any  material  management 
control  we^esses,  we  did  not  assess  management's  selfrevaluation. 

Adequacy  of  Management  Controls.  The  DoD  Components'  management 
controls  over  the  coordination,  dissemination,  and  use  of  foreign  materiel 
exploitation  results  were  adequate;  we  identified  no  material  management 
control  weaknesses.  We  did  not  consider  the  Military  Departments'  threat 
validation  issues  to  be  material  management  control  weaknesses. 
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Summary  of  Prior  Coverage 


Office  of  the  Inspector  General,  DoD,  Audit  Report  No.  97-133,  "Acquisition 
and  Exploitation  of  Foreign  Materiel  by  DoD  Components,"  Apffi  21,  1997, 
identifies  several  positive  developments  within  the  Foreign  Materiel  Program. 
The  Office  of  the  Secretary  of  Defense,  the  Defense  Agencies,  ^d  the  Military 
Departments  were  developing  new  foreign  materiel  acquisition  prc^ss 
procedures  to  improve  competition  when  awarding  foreign  materiel  acquisition 
contracts.  In  admtion,  Ae  DoD  Components  solicited  exploitation  requirements 
from  DoD  customers  for  incorporation  into  exploitation  plans.  However,  the 
report  also  points  out  issues  that  the  DoD  Components  must  address  to  ensure  a 
more  efficient  and  effective  foreign  materiel  program.  Those  issues  included 
enhancements  to  the  foreign  materiel  acquisition  process,  support  of  test  Md 
evaluation  requirements,  and  development  of  a  complete  and  accurate  foreign 
materiel  inventory  data  base. 


13 


Appendix  B.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 
Director,  Defense  Test,  Systems  Bigineering,  and  Evaluation 
Deputy  Director,  Test  Facility  Resources 
Director,  Defense  Logistics  Studies  Information  Exchange 
Director,  Defense  Meeting  and  Simulation  Office 
Under  Secretz^  of  Defense  (Comptroller) 

Dq)uty  Chief  Financial  Officer 
Dq)uty  Comptroller  (Program/Budget) 

Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and  fiitelligence) 
Deputy  Assistant  Secretary  of  Defense  (Intelligence  and  Security) 

Director,  Office  for  Oversight  and  Sp^ial  Technology 
Assistant  Secretaty  of  Defense  (Public  Affairs) 

Director,  Operational  Test  and  Evaluation 


Department  of  the  Army 

Commander,  Army  Materiel  Command 

Commander,  Simulation,  Training,  and  Instrumentation  Command 
Project  Manager-Instrumentation,  Targets,  and  Threat  Simulators 
Director,  Targets  Managemait  Office 
Director,  Threat  Simulator  hfenagement  Office 
Assistant  Secret^  of  the  Army  (FinancM  Management  and  Comptroller) 
Assistant  Secretary  of  the  Army  (Research,  Development,  and  Acquisition) 
Deputy  Chief  of  Staff  for  Intelligence 
Auditor  General,  Department  of  the  Army 
Commander,  Army  Operational  Test  and  Evaluation  Command 
Commander,  Army  Intelligence  and  Security  Command 
Commander,  National  Ground  Intelligence  Center 
Director,  Army  Test  and  Evaluation  Management  Agency 
Director,  Army  Modeling  and  Simulation  Office 
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Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management  and  Comptroller) 
Assistant  Secretary  of  the  Navy  (Research,  Developmait,  and  Acquisition) 
Auditor  General,  Department  of  the  Navy 
Director,  Office  of  Naval  Intelligence 
Director,  Space,  Information  Warfare,  Command  and  Control 
Director,  Navy  Modeling  and  Simulation  Management  Office 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Acquisition) 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Air  Force 
Director,  Air  Force  Test  and  Evaluation  Directorate 
Chief,  Foreign  Materiel  Office 
Director,  Command  and  Control  Directorate 

Chief,  Air  Force  Modeling,  Simulation,  and  Analysis  (Policy  and  Standards 
Division) 

Commander,  National  Air  Intelligrace  Center 


Other  Defense  Organizations 

Director,  Defense  Contr^t  Audit  Agency 
Director,  Defense  Logistics  Agency 
Director,  Defense  Intelligence  Agency 

Director,  Missile  and  Space  Intelligence  Center 
Director,  Threat  Systems  Office 
Inspector  General,  Defense  Intelligence  Agency 
Director,  National  Security  Agency 

Inspector  General,  National  Security  Agency 
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Non-Defense  Federal  Organizations  and  Individuals 

Office  of  Management  and  Budget 

Technical  Information  Center,  National  Security  and  International  Affairs  Division, 
General  Accounting  Office 

Chflirman  and  ranking  minority  member  of  each  of  the  following  congressional 
committees  and  subcommittees: 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 

Senate  Committee  on  Armed  Services 

Senate  Committee  on  Government  Affairs 

Senate  Select  Committee  on  Intelligence 

House  Committee  on  Appropriations 

House  Subcommittee  on  National  Security,  Committee  on  Appropriations 
House  Committee  on  Government  Reform  and  Oversight 
House  Subcommittee  on  Government  Management,  Information,  and  Technology, 
Committee  on  Government  Reform  and  Oversight 
House  Subcommittee  on  National  Security,  International  Affairs,  and  Criminal 
Justice,  Committee  on  Government  Reform  and  Oversight 
House  Committee  on  National  Security 
House  Permanent  Select  Committee  on  Intelligence 
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Part  ni  -  Management  Comments 
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Department  of  the  Navy  Comments 


DEPARTMENT  OF  THE  NAVY 

OFFICE  OF  THE  CHIEF  OF  NAVAL  OFERATIONS 
aOOO  NAVY  FEHTAaON 
WASHINOTON.  DO.  a03H-M00 

IN  REPLY  REFER  TO 

3800 

Ser  N2/7D550764 
10  Sep  97 

From:  Chief  of  Navcd  Operations  (N2) 

TO:  Inspector  General,  Department  of  Defense 

Subj:  AUDIT  REPORT  ON  USB  OF  FOREICar  MATERIEL  EXPLOITATION 
RESULTS  (PROJECT  HUMBER  7AD-0008) 

Ref:  (a)  DODI6  Memo  of  30  Jun  97 

(b)  SECNAVINST  5200.38 

1.  In  response  to  reference  (a) ,  concur  with  the  findings  and 
recommendations  of  the  subject  audit. 

2.  The  Chief  of  Naval  Operations  (N2}  and  the  Intelligence 
Directorate  of  the  Office  of  Naval  Intelligence  (ONI'2)  will 
assist  the  Director,  Navy  Modeling  and  Simulation  Management 
Office  in  the  development  of  the  Navy- vide  validation  process,  in 
accordance  with  reference  (b) . 


JlAMUjdL 


P.  M.  LOHELL 

Director  of  Naval  Intelligence 
Acting 


Department  of  the  Air  Force  Comments 


Final 

Mwww  . 


DEPAimiENTOFTHE  AIR  FORCE 

WADOIMirrEHS  UMTED  STATES  AM  nncc 
WASHWOTONOe 


SEP  2  1997 


MEMORANDUM  FOB  ASSISTANTINSPECTOB  GENERAL  FOR  AUDITING 
OFFICE  OF  THE  INSPECTOR  GENERAL 
DEPARTMENT  OF  DEFENSE 

FROM:  AF/XOC 

1480  Air  Force  Pentagon 
Washington,  DC  20330-1480 

SUBJECT:  Audit  Report  on  Use  of  Foreign  Material  Exploitation  Reaulta  (Pnyect 
No.  7AD-006) 


This  is  in  reply  to  your  memorandum  requesting  the  Assistant  Secretary  of 
the  Air  Force  (Financial  Management  and  Comptroller)  to  inrovide  comments  on  tiie 
dted  report.  Your  acknowledgment  of  the  Air  Force  initiative  to  partner  mth  the 
dectronic  countermeasures  vendor  community  is  noteworthy  and  will  aid  in 
accelerating  DoD-wide  acceptance. 

We  non  concur  with  the  draft  report  as  written  because  the  second  finding 
(page  9)  does  not  take  into  account  our  current  efforts  to  address  this  problem. 
Although  not  yet  published,  the  Air  Force  is  concluding  a  seven  month  effort  to 
revise  Air  Force  Instruction  (AFI)  16-1001  in  which  the  criteria  you  suggest  has 
already  been  developed.  Attachedisacopyofthe  draft  AFI,  and  the  Air  Force  can 
provide  you  a  finalized  copy  once  it  is  coordinated  &nd  published  by  the  end  of  the 
calendar  year. 

Questions  should  be  addressed  to  Mr  Raymond  Miller,  HQ  USAF/KOCP,  at 
commercial  (202)  761-5341,  ext  102  or  DSN  763-6341,  ext  102. 


Pages 

Revised 


^^MES  E.  SANDSTBOM,  Brig  Gen,  USAF 
Director,  Command  and  Ciontrol 
DCS,  Tnkna  and  Operationa 


Attachment: 

AFI  16-1001  (Draft) 


♦Omitted  because  of  length.  Copies  will  be  provided  upon  request 
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